A search engine is a tool that identifies documents, typically stored on hosts distributed over a network, which satisfy search queries specified by users. Web search engines work by storing information about a large number of documents (such as web pages) which they retrieve from the World Wide Web (WWW) via a web crawler. Crawling a website during high traffic periods and/or excessive load during crawling can deplete network resources from the website, rendering the website less accessible to others. On the other hand, it may be that not enough crawling is performed for the website. Thus, it may be advantageous to enable a website owner to control the rate at which the website is crawled.
Like reference numerals refer to corresponding parts throughout the drawings.